Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate: support migration of FBC to latest preferred FBC #1144

Merged
merged 2 commits into from
Oct 10, 2023

Conversation

joelanford
Copy link
Member

This commit adds support for migrating FBC to the latest preferred FBC
contents. Note that sqlite and bundle inputs are always rendered using
the latest preferred FBC contents.

The migrate command is updated to now support FBC images and directories
as input (only sqlite was supported prior), such that the written output
will always be migrated.

The render command is updated with a --migrate flag that allows a
caller to opt into migration during rendering.

Under the hood, both of these subcommands use the action.Render struct,
which has a new Migrate boolean field that callers can use to
enable/disable the migration behavior.

Signed-off-by: Joe Lanford joe.lanford@gmail.com


This is specifically useful right now to convert existing FBCs using olm.bundle.object to use of olm.csv.metadata, which can reduce catalog size significantly. At the time of the PR being opened, it reduces the operatorhub catalog from 98M to 23M (~76% reduction in size)

NOTE: This builds upon #1143, which is necessary to align on the tenet that FBC directory structure can change without changing the validity or meaning of the catalog.

Reviewer Checklist

  • Implementation matches the proposed design, or proposal is updated to match implementation
  • Sufficient unit test coverage
  • Sufficient end-to-end test coverage
  • Docs updated or added to /docs
  • Commit messages sensible and descriptive

This commit removes a feature of FBC that has never actually been used
in practice: the ability to reference a file in an olm.bundle.object
property, where the path is relative to the directory in which the file
containing the olm.bundle.object property exists. This was originally
intended to be a way to avoid bloating the FBC, but it's presence has
caused two problems:

  1. It hinted that it would be okay for third-party properties and
     schemas to reference files in a similar way.
  2. Because of (1), we have never really been able to make assumptions
     that would enable us to migrate and re-write FBC in a different
     hierarchy, which has been limiting.

In short, it imposes a burden on catalog maintainers to keep a catalog
in a filesystem structure that is imposed by the author of the catalog
contribution.

In practice, ref-style olm.bundle.object properties have never been used
(as far as I'm aware), because no tooling has ever produced that style,
and no one I have heard of is using other methods to render bundles into
an FBC.

Lastly, with the recent addition of the `olm.csv.metadata` property,
the useful life of the `olm.bundle.object` property (which has always
been alpha) is nearing an end.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
This commit adds support for migrating FBC to the latest preferred FBC
contents. Note that sqlite and bundle inputs are always rendered using
the latest preferred FBC contents.

The migrate command is updated to now support FBC images and directories
as input (only sqlite was supported prior), such that the written output
will always be migrated.

The render command is updated with a `--migrate` flag that allows a
caller to opt into migration during rendering.

Under the hood, both of these subcommands use the action.Render struct,
which has a new `Migrate` boolean field that callers can use to
enable/disable the migration behavior.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
@joelanford joelanford added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 22, 2023
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 22, 2023
@openshift-ci openshift-ci bot requested review from exdx and jmrodri September 22, 2023 19:01
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 22, 2023
@codecov
Copy link

codecov bot commented Sep 22, 2023

Codecov Report

Attention: 28 lines in your changes are missing coverage. Please review.

Comparison is base (d02b0b6) 53.79% compared to head (22e30c9) 53.83%.
Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1144      +/-   ##
==========================================
+ Coverage   53.79%   53.83%   +0.03%     
==========================================
  Files         108      108              
  Lines       10199    10224      +25     
==========================================
+ Hits         5487     5504      +17     
- Misses       3744     3746       +2     
- Partials      968      974       +6     
Files Coverage Δ
alpha/action/migrate.go 73.33% <100.00%> (ø)
alpha/property/property.go 75.60% <100.00%> (-3.85%) ⬇️
pkg/api/api_to_model.go 69.47% <0.00%> (ø)
pkg/registry/registry_to_model.go 61.53% <0.00%> (ø)
alpha/declcfg/load.go 77.77% <50.00%> (+3.07%) ⬆️
alpha/action/render.go 67.35% <69.23%> (+0.68%) ⬆️
pkg/cache/json.go 59.18% <70.00%> (+2.15%) ⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 25, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: everettraven, joelanford

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cdjohnson
Copy link
Contributor

@joelanford Thanks for this PR. I think this will be very useful.

I did some testing, and I noticed that you may have inadvertently changed the default for rendering a bundle image. Is this on purpose? I would think that the --migrate option also applied to a bundle image reference.

img="icr.io/cpopen/ibm-mq-operator-bundle@sha256:11639e73211324978bde65fbe35d7f71ee2b7efd96e4d5d1c8158855256c2cae"

opmnew=~/tools/opm/dev/opm
opmold=~/tools/opm/1.26.1/opm
fbcdir=./configs/ibm-mq

# Renders olm.bundle.properties - OK
$opmold render $img > bundle-render-old.yaml

# Renders olm.csv.metadata - ERROR
$opmnew render $img > bundle-render-new.yaml

# Renders olm.csv.metadata - OK
$opmnew render --migrate $img  > bundle-render-migrate-new.yaml

# Renders olm.bundle.properties - OK
$opmold render $fbcdir > catalog-render-old.yaml

# Renders olm.bundle.properties - OK
$opmnew render $fbcdir > catalog-render-new.yaml

# Renders olm.csv.metadata - OK
$opmnew render --migrate $fbcdir > catalog-render-migrate-new.yaml

@joelanford
Copy link
Member Author

# Renders olm.csv.metadata - ERROR
$opmnew render $img > bundle-render-new.yaml

You're saying that this command produces olm.csv.metadata even without --migrate=true? That's the behavior I intended. That has been the case since 1.28+.

The --migrate flag is a no-op when the inputs to render are sqlite DBs and bundles. The reasoning is essentially that opm should be converting sqlite DBs and bundles directly to the latest/best formats, rather than doing the extra work of converting them to the oldest format and then running all of the migrations.

@cdjohnson
Copy link
Contributor

You're saying that this command produces olm.csv.metadata even without --migrate=true? That's the behavior I intended. That has been the case since 1.28+.

Yes, I didn't expect that. I happened to be testing with 1.26.1. I retested with 1.29.0, and you're right, and it's clear in that PR. Just missed it.

@grokspawn
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 10, 2023
@openshift-ci openshift-ci bot merged commit 5e3fa99 into operator-framework:master Oct 10, 2023
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants